Perspective-correct communication window with motion parallax

ABSTRACT

A perspective-correct communication window system and method for communicating between participants in an online meeting, where the participants are not in the same physical locations. Embodiments of the system and method provide an in-person communications experience by changing virtual viewpoint for the participants when they are viewing the online meeting. The participant sees a different perspective displayed on a monitor based on the location of the participant&#39;s eyes. Embodiments of the system and method include a capture and creation component that is used to capture visual data about each participant and create a realistic geometric proxy from the data. A scene geometry component is used to create a virtual scene geometry that mimics the arrangement of an in-person meeting. A virtual viewpoint component displays the changing virtual viewpoint to the viewer and can add perceived depth using motion parallax.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of a prior application entitle“PERSPECTIVE-CORRECT COMMUNICATION WINDOW WITH MOTION PARALLAX”, whichwas assigned Ser. No. 13/602,097 and filed Aug. 31, 2012, and whichclaimed the benefit of and priority to Provisional U.S. PatentApplication Ser. No. 61/653,983, filed May 31, 2012.

BACKGROUND

Current video conferencing technology typically uses a single camera tocapture RGB data (from the red, blue, and green (RGB) color model) of alocal scene. This local scene typically includes the people that areparticipating in the video conference, or meeting participants. The datathen is transmitted in real time to a remote location and then displayedto another meeting participant that is in a different location than theother meeting participant.

While advances have been made in video conferencing technology that helpprovide a higher definition capture, compression, and transmission,typically the experience falls short of recreating the face-to-faceexperience of an in-person conference. One reason for this is that thetypical video conferencing experience lacks eye gaze and other correctconversational geometry. For example, typically the person beingcaptured remotely is not looking into your eyes, as one would experiencein a face-to-face conversation. This is because their eyes are notlooking where the camera is located and instead are looking at thescreen. Moreover, three-dimensional (3D) elements like motion parallaxand image depth, as well as the freedom to change perspective in thescene are lacking because there is only a single, fixed video cameracapturing the scene and the meeting participants.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter.

Embodiments of the perspective-correct communication window system andmethod provide a way to create an in-person communications experiencefor participants in an online meeting or conference. Embodiments of thesystem and method provide a correct perspective of the participantsusing a changing virtual viewpoint for the participants when they areviewing the online meeting. This changing virtual viewpoint is dependenton the position and orientation of the viewer's face or moreparticularly the viewer's eyes. Depending on the position andorientation of the face, the viewer sees a different perspective of theother participants in the meeting as well as the background in thedisplay.

Embodiments of the system and method generally include three components.A first component, the capture and creation component, is used tocapture visual data about each participant and create a realisticgeometric proxy from the data. This geometric proxy is a geometricrepresentation of the participant that has real video painted onto thegeometric representation frame by frame in order to increase therealism. Moreover, a geometric proxy is created for each participant inthe meeting. The data is captured using one or more camera pods. In someembodiments these camera pods include a stereoscopic infrared (IR)camera and an IR emitter (to capture depth information) and a RGB camera(to capture RGB data). The camera pod layout at each endpoint isvariable and dependent on the number of camera pods available at theendpoint. Each geometric proxy is created using the RGD data and thedepth information.

A second component is a scene geometry component, which is used tocreate a virtual scene geometry that imitates the arrangement of anin-person meeting. The scene geometry is dependent on the number ofparticipants in the meeting. Creating the scene geometry includes boththe registration of the three-dimensional (3D) volume and the alignmentof the 3D space that the camera pods capture. The general idea of thescene geometry component is to create relative geometry between themeeting participants. The scene is aligned virtually to mimic areal-life scene as if the participants are in the same physical locationand engaged in an in-person conversation.

The scene geometry uses virtual boxes to have relative, consistentgeometry between the participants. A meeting with two participants (or aone-to-one (1:1) scene geometry) consists of two boxes that occupy thespaces in front of the respective monitors (not shown) of the twoparticipants. When there are three participants the scene geometryincludes three virtual boxes that are placed around a virtual roundtable in an equidistant manner.

The scene geometry also includes a virtual camera. The virtual camera isa composition of images from two or more of the camera pods in order toobtain a camera view that is not captured by any one camera pod alone.This allows embodiments of the system and method to obtain a natural eyegaze and connection between people. Face tracking (or more specifically,eye tracking), is used to improve performance by helping the virtualcamera remain level with the eye gaze of the viewer. In other words theface tracking provides a correct virtual camera view that is alignedwith the viewer's eyes. This means that the virtual camera interactswith the face tracking to create a virtual viewpoint that has the userlooking straight at the other participant.

Each geometric proxy is rendered relative to each other in the scenegeometry. The rendered geometric proxies and scene geometry is thentransmitted to each of the participants. The third component is thevirtual viewpoint component, which displays a changing virtual viewpointto the viewer based on the position and orientation of the viewer'sface. This motion parallax effect adds realism to the scene displayed onthe monitor. In addition, face tracking can be used to track theposition and orientation of the viewer's face. What the viewer sees onthe monitor in one facial position and orientation is different fromwhat the viewer sees in another facial position and orientation.

Embodiments of the system and method also include facilitating multipleparticipants at a single endpoint. An endpoint means a location orenvironment containing one or more participants of the conference ormeeting. In some embodiments a face tracking technique tracks twodifferent faces and then provides different views to different viewers.In other embodiments glasses are worn by each of the multipleparticipants at the endpoint and in some embodiments the glasses haveactive shutters on them that show each wearer alternating framesdisplayed by the monitor that are tuned to each pair of glasses. Otherembodiments use a monitor having multiple viewing angles such that aviewer looking at the monitor from the right side sees one scene andanother viewer looking at the monitor from the left sees a differentscene.

It should be noted that alternative embodiments are possible, and stepsand elements discussed herein may be changed, added, or eliminated,depending on the particular embodiment. These alternative embodimentsinclude alternative steps and alternative elements that may be used, andstructural changes that may be made, without departing from the scope ofthe invention.

DRAWINGS DESCRIPTION

Referring now to the drawings in which like reference numbers representcorresponding parts throughout:

FIG. 1 is a block diagram illustrating a general overview of embodimentsof the perspective-correct communication window system and methodimplemented in a computing environment.

FIG. 2 is a block diagram illustrating the system details of embodimentsof the perspective-correct communication window system and method shownin FIG. 1.

FIG. 3 illustrates a simplified example of a general-purpose computersystem on which various embodiments and elements of theperspective-correct communication window system and method, as describedherein and shown in FIGS. 1, 2, and 4-8, may be implemented.

FIG. 4 is a flow diagram illustrating the general operation ofembodiments of the perspective-correct communication window system andmethod shown in FIGS. 1 and 2.

FIG. 5 is a block diagram illustrating the details of an exemplaryembodiment of a camera pod of embodiments of the perspective-correctcommunication window system and method shown in FIG. 1.

FIG. 6 illustrates an exemplary embodiment of a camera pod layout (suchas that shown in FIG. 2) using four camera pods.

FIG. 7 illustrates an exemplary overview of the creation of a geometricproxy for a single meeting participant.

FIG. 8 illustrates an exemplary embodiment of the scene geometry betweenparticipants when there are two participants in the meeting (a 1:1conference).

FIG. 9 illustrates an exemplary embodiment of the scene geometry betweenparticipants when there are three participants in the meeting (a3-endpoint conference).

FIG. 10 illustrates an exemplary embodiment of a virtual camera based onwhere the participant is looking.

FIG. 11 illustrates an exemplary embodiment of providing depth throughmotion parallax based on where a viewer is facing.

FIG. 12 illustrates an exemplary embodiment of a technique to handlemultiple participants at a single endpoint.

DETAILED DESCRIPTION

In the following description of perspective-correct communication windowsystem and method reference is made to the accompanying drawings, whichform a part thereof, and in which is shown by way of illustration aspecific example whereby embodiments of the perspective-correctcommunication window system and method may be practiced. It is to beunderstood that other embodiments may be utilized and structural changesmay be made without departing from the scope of the claimed subjectmatter.

I. System Overview

Embodiments of the perspective-correct communication window system andmethod provide a way provide an “in person” communications experiencefor users. FIG. 1 is a block diagram illustrating a general overview ofembodiments of the perspective-correct communication window system 100and method implemented in a computing environment. In particular,embodiments of the system 100 and method are implemented on a computingdevice 110. This computing device may be a single computing device ormay be spread out over a plurality of devices. Moreover, the computingdevice 110 may be virtually any device having a processor, including adesktop computer, a tablet computing device, and an embedded computingdevice.

As shown in FIG. 1, the computing environment includes a firstenvironment 120 and a second environment 125. In the first environment120, a first participant 130 is captured by a plurality of first camerapods 135. It should be noted that four camera pods are shown in FIG. 1,but fewer or more camera pods can be used. Also as shown in FIG. 1, thefirst plurality of camera pods 135 are shown attached to a first monitor140, which is in communication with the computing device 110. However,it should be noted that in alternate embodiments the first plurality ofcamera pods 135 may be mounted on some other structure or there may besome mounted on the first monitor 140 and others mounted on otherstructures.

The first participant 130 is captured by the first plurality of camerapods 135 and processed by embodiments of the perspective-correctcommunication window system 100 and method, as explained in detailbelow. This processed information is transmitted across a network 150using a first communication link 155 (from the first environment 120 tothe network 150) and a second communication link 160 (from the network150 to the second environment 125. In FIG. 1 embodiments of the system100 and method are shown residing on the network 150. However, it shouldbe noted that this is only one way in which the system 100 and methodmay be implemented.

The transmitted processed information is received in the secondenvironment 125, processed by embodiments of the system 100 and method,and then displayed to a second participant 170 on a second monitor 175.As shown in FIG. 1, the second monitor 175 contains a second pluralityof camera pods 180 that are used to capture the second participant 170.In addition, the second plurality of camera pods 180 are used to trackthe eye gaze of the second participant 170 and determine how theprocessed information should be presented to the second participant 170.This is explained in more detail below. Moreover, the first plurality ofcamera pods 135 is also are used to track the eye gaze of the firstparticipant 130 and determine how processed information should bepresented to the first participant 130. In alternate embodiments eyegaze is tracked using some other device that a camera pod, such as anexternal camera.

It should be noted that embodiments of the system 100 and method work inboth directions. In other words, the first environment 120 can alsoreceive transmissions from the second environment 125 and the secondenvironment 125 can also transmit processed information. For pedagogicalpurposes, however, only the transmission from the first environment 120to the system 120 and method and on to the second environment 125 isdiscussed above.

II. System Details

Embodiments of the system 100 and method include three main componentsthat work together to create that “in person” communications experience.The first component is capturing and creating a three-dimensional (3D)video image of each person participating in the conference. The secondcomponent is creating the relevant scene geometry based on the number ofparticipants in the conference. This component ensures that theresultant geometry between virtual viewpoints (or windows) at theendpoints is the same. And the third component is rendering andproviding a virtual view as if the camera was positioned from theperspective of where the viewer is looking, thereby recreating the samescene geometry participants would have when talking in person.

FIG. 2 is a block diagram illustrating the system details of embodimentsof the perspective-correct communication window system 100 and methodshown in FIG. 1. As shown in FIG. 2, embodiments of the system 100 andmethod include a capture and creation component 200, a scene geometrycomponent 210, and a virtual viewpoint component 220. The capture andcreation component is used for capturing and creating a 3D video imageof the participant.

Specifically, the capture and creation component 200 includes a camerapod layout 230 that includes a plurality of camera pods. The camera podlayout 230 is used to capture a participant from multiple perspectives.Computer vision methods are used to create a high-fidelity geometryproxy for each meeting participant. As explained in detail below, thisis achieved by taking RBG data obtained from an RGB data collectionmodule 235 and depth information obtained and computed by a depthinformation computation module 240. From this information a geometricproxy creation module 245 creates a geometric proxy 250 for eachparticipant. Image-based rendering methods are used to createphotorealistic textures for the geometric proxy 250 such as withview-dependent texture mapping.

The scene geometry component 210 is used to create the correct scenegeometry to simulate participants being together in a real conversation.This scene geometry is dependent on the number of participants (orendpoints) in the conference. A 3D registration module 260 is used toobtain a precise registration of a monitor with the camera pods.Moreover, a space alignment module 265 aligns the orientation of thecamera pods with the real world. For a 1:1 meeting (having twoparticipants), this is simply the two physical spaces lined up acrossfrom one another in the virtual environment. The capture area that isbeing recreated for each participant is the area in front of themonitor.

Once the textured geometric proxy 250 has been created for each meetingparticipant and the participants are represented in a 3D virtual spacethat is related to the other participants in the conference, thegeometric proxies are rendered to each other in a manner consistent withconversational geometry. Moreover, this rendering is done based on thenumber of participants in the conference. Virtual boxes are used toensure that an eye gaze and conversational geometry between theparticipants is correct and that to other participants theconversational geometry looks correct so that the viewer can correctlysee the other participants.

The geometric proxies and in some cases the registration and alignmentinformation are transmitted to remote participants by the transmissionmodule 270. The virtual viewpoint component 220 is used to enhance thevirtual view rendered to the remote participants. The experience of‘being there’ is enhanced through the use of a motion parallax module280 that adds motion parallax and depth to the scene behind theparticipants. Horizontal and lateral movements by either participantchange the viewpoint shown on their local displays and the participantsees the scene they are viewing, and the person in it, from a differentperspective. This greatly enhances the experience of the meetingparticipants.

III. Exemplary Operating Environment

Before proceeding further with the operational overview and details ofembodiments of the perspective-correct communication window system andmethod, a discussion will now be presented of an exemplary operatingenvironment in which embodiments of the perspective-correctcommunication window system 100 and method may operate. Embodiments ofthe perspective-correct communication window system 100 and methoddescribed herein are operational within numerous types of generalpurpose or special purpose computing system environments orconfigurations.

FIG. 3 illustrates a simplified example of a general-purpose computersystem on which various embodiments and elements of theperspective-correct communication window system 100 and method, asdescribed herein and shown in FIGS. 1, 2, and 4-12, may be implemented.It should be noted that any boxes that are represented by broken ordashed lines in FIG. 3 represent alternate embodiments of the simplifiedcomputing device, and that any or all of these alternate embodiments, asdescribed below, may be used in combination with other alternateembodiments that are described throughout this document.

For example, FIG. 3 shows a general system diagram showing a simplifiedcomputing device 10. The simplified computing device 10 may be asimplified version of the computing device 110 shown in FIG. 1. Suchcomputing devices can be typically be found in devices having at leastsome minimum computational capability, including, but not limited to,personal computers, server computers, hand-held computing devices,laptop or mobile computers, communications devices such as cell phonesand PDA's, multiprocessor systems, microprocessor-based systems, set topboxes, programmable consumer electronics, network PCs, minicomputers,mainframe computers, audio or video media players, etc.

To allow a device to implement embodiments of the perspective-correctcommunication window system 100 and method described herein, the deviceshould have a sufficient computational capability and system memory toenable basic computational operations. In particular, as illustrated byFIG. 3, the computational capability is generally illustrated by one ormore processing unit(s) 12, and may also include one or more GPUs 14,either or both in communication with system memory 16. Note that thatthe processing unit(s) 12 of the general computing device of may bespecialized microprocessors, such as a DSP, a VLIW, or othermicro-controller, or can be conventional CPUs having one or moreprocessing cores, including specialized GPU-based cores in a multi-coreCPU.

In addition, the simplified computing device 10 of FIG. 3 may alsoinclude other components, such as, for example, a communicationsinterface 18. The simplified computing device 10 of FIG. 3 may alsoinclude one or more conventional computer input devices 20 (such asstyli, pointing devices, keyboards, audio input devices, video inputdevices, haptic input devices, devices for receiving wired or wirelessdata transmissions, etc.). The simplified computing device 10 of FIG. 3may also include other optional components, such as, for example, one ormore conventional computer output devices 22 (e.g., display device(s)24, audio output devices, video output devices, devices for transmittingwired or wireless data transmissions, etc.). Note that typicalcommunications interfaces 18, input devices 20, output devices 22, andstorage devices 26 for general-purpose computers are well known to thoseskilled in the art, and will not be described in detail herein.

The simplified computing device 10 of FIG. 3 may also include a varietyof computer readable media. Computer readable media can be any availablemedia that can be accessed by the simplified computing device 10 viastorage devices 26 and includes both volatile and nonvolatile media thatis either removable 28 and/or non-removable 30, for storage ofinformation such as computer-readable or computer-executableinstructions, data structures, program modules, or other data. By way ofexample, and not limitation, computer readable media may comprisecomputer storage media and communication media. Computer storage mediaincludes, but is not limited to, computer or machine readable media orstorage devices such as DVD's, CD's, floppy disks, tape drives, harddrives, optical drives, solid state memory devices, RAM, ROM, EEPROM,flash memory or other memory technology, magnetic cassettes, magnetictapes, magnetic disk storage, or other magnetic storage devices, or anyother device which can be used to store the desired information andwhich can be accessed by one or more computing devices.

Retention of information such as computer-readable orcomputer-executable instructions, data structures, program modules,etc., can also be accomplished by using any of a variety of theaforementioned communication media to encode one or more modulated datasignals or carrier waves, or other transport mechanisms orcommunications protocols, and includes any wired or wireless informationdelivery mechanism. Note that the terms “modulated data signal” or“carrier wave” generally refer to a signal that has one or more of itscharacteristics set or changed in such a manner as to encode informationin the signal. For example, communication media includes wired mediasuch as a wired network or direct-wired connection carrying one or moremodulated data signals, and wireless media such as acoustic, RF,infrared, laser, and other wireless media for transmitting and/orreceiving one or more modulated data signals or carrier waves.Combinations of the any of the above should also be included within thescope of communication media.

Further, software, programs, and/or computer program products embodyingthe some or all of the various embodiments of the perspective-correctcommunication window system 100 and method described herein, or portionsthereof, may be stored, received, transmitted, or read from any desiredcombination of computer or machine readable media or storage devices andcommunication media in the form of computer executable instructions orother data structures.

Finally, embodiments of the perspective-correct communication window 100and method described herein may be further described in the generalcontext of computer-executable instructions, such as program modules,being executed by a computing device. Generally, program modules includeroutines, programs, objects, components, data structures, etc., thatperform particular tasks or implement particular abstract data types.The embodiments described herein may also be practiced in distributedcomputing environments where tasks are performed by one or more remoteprocessing devices, or within a cloud of one or more devices, that arelinked through one or more communications networks. In a distributedcomputing environment, program modules may be located in both local andremote computer storage media including media storage devices. Stillfurther, the aforementioned instructions may be implemented, in part orin whole, as hardware logic circuits, which may or may not include aprocessor.

IV. Operational Overview

FIG. 4 is a flow diagram illustrating the general operation ofembodiments of the perspective-correct communication window system 100and method shown in FIGS. 1 and 2. As shown in FIG. 3, the operation ofembodiments of the perspective-correct communication window system 100and method begins by capturing images of each of the participants in theconference or meeting (box 400). At least one of the participants is aremote participant, which means that the remote participant is not inthe same physical location as the other participant. The capture of eachparticipant is achieved by using the camera pods.

Next, embodiments of the method use data from the captured images tocreate a geometric proxy for each participant (box 410). The number ofparticipants then is determined (box 420). This determination may beperformed out of order such that the number of participants isdetermined or known beforehand. Embodiments of the method then generatescene geometry based on the number of participants (box 430). This scenegeometry generation helps to simulate the experience of an in-personconversation or meeting with the remote participants.

Each geometric proxy for a particular participant then is rendered tothe other geometric proxies for the other participants within the scenegeometry (box 440). This rendering is performed such that the geometricproxies are arranged in a manner that is consistent with an in-personconversation. These rendered geometric proxies and the scene geometrythen are transmitted to the participants (box 450). A changing virtualviewpoint is displayed to each of the participants such that the virtualviewpoint is dependent on an orientation of the viewer's face (box 460).For additional realism, motion parallax and depth are added in order toenhance the viewing experience for the participants (box 470). Asexplained in detail below, the motion parallax and depth are dependenton the eye gaze of the participant relative to the monitor on which theparticipant in viewing the conference or meeting.

V. Operational Details

The operational details of embodiments of the perspective-correctcommunication window system 100 and method will now be discussed. Thisincludes the details of the camera pods, camera pod layout, thegeometric proxy creation, and the creation of the scene geometry.Moreover, also discussed will be the concept of a virtual camera, theaddition of motion parallax and depth to the geometric proxies and scenegeometry, and the handling of more than one participant in the sameenvironment and viewing the same monitor.

V.A. Camera Pod

The first component of embodiments of the perspective-correctcommunication window system 100 and method is the capture and creationcomponent 200. This component includes a plurality of camera pods thatare used to capture the 3D scene. Moreover, as explained below, eachcamera pod contains multiple sensors.

FIG. 5 is a block diagram illustrating the details of an exemplaryembodiment of a camera pod 500 of embodiments of the perspective-correctcommunication window system 100 and method shown in FIG. 1. As notedabove, embodiments of the system 100 and method typically include morethan one camera pod 500. However, for pedagogical purposes only a singlecamera pod will be described. Moreover, it should be noted that themultiple camera pods do not necessarily have to include the samesensors. Some embodiments of the system 100 and method may include aplurality of camera pods that contain different sensors from each other.

As shown in FIG. 5, the camera pod 500 includes multiple camera sensors.These sensors include stereoscopic sensors infrared (IR) cameras 510, anRGB camera 520, and an IR emitter 530. In order to capture a 3D image ofthe scene the camera pod 500 captures RGB data and the depth coordinatesin order to compute a depth map. FIG. 5 illustrates that the IRstereoscopic IR cameras 510 and the IR emitter 530 are used to capturethe depth calculation. The RGB camera 520 is used for the textureacquisition and to reinforce the depth cues using depth segmentation.Depth segmentation, which is well known in the computer vision field,seeks to separate objects in an image from the background usingbackground subtraction.

In alternative embodiments, the camera pod 500 achieves stereoscopicsensing using time of flight sensors or ultrasound instead of the IRstructure light approach. A time-of-flight camera is a range imagingcamera system that computes distance based on the speed of light and bymeasuring the time of flight of a light signal between the camera andthe object for each point in an image. Ultrasound techniques can be usedcompute distance by generating an ultrasonic pulse in a certaindirection. If there is an object in the path of the pulse, then part orall of the pulse will be reflected back to the transmitter as an echo.The range can be found by measuring the difference between the pulsebeing transmitted and the echo being received. In other embodiments thedistance may be found be performing an RGB depth calculation usingstereo pairs of RGB camera.

V.B. Camera Pod Layout

Also part of the capture and creation component 200 is the camera podlayout. One or more camera pods are configured in a particular layout inorder to capture the 3D scene that includes one or more of theparticipants. The number of camera pods directly affects the quality ofthe captured images and the number of occlusions. As the number ofcamera pods increases there is more RGB data available and this improvesimage quality. Moreover, the number of occlusions is diminished as thenumber of camera pods increases.

In some embodiments of the system 100 and method the camera pod layoutuses four camera pods. In alternate embodiments any number of camerasmay be used. In fact there could be a lower-end version that uses asingle camera pod. For example, the single camera pod may be mounted ontop of a monitor and use image distortion correction techniques tocorrect for any imaging errors. The touchstone is that the camera podlayout should have enough camera pods to provide a 3D view of theenvironment containing the participant.

FIG. 6 illustrates an exemplary embodiment of a camera pod layout (suchas that shown in FIG. 2) using four camera pods. As shown in FIG. 6, thefour camera pods 500 are embedded in the bezel of a monitor 600. Themonitor 600 can be of virtually any size, but larger monitors provide amore life-size re-projection. This typically provides the user with themore realistic experience. Displayed on the monitor 600 is a remoteparticipant 610 that is participating in the conference or meeting.

As shown in FIG. 6, the four camera pods 500 are arranged in a diamondconfiguration. This allows embodiments of the system 100 and method tocapture the user from above and below and from side to side. Moreover,the two middle top and bottom camera pods can be used to get a realistictexture on the face of the user without a seam. Note that cameras in thecorners will typically causes a seam issue. In other embodimentsvirtually any configuration and arrangement of the four camera pods 500can be used and may be mounted anywhere on the monitor 600. In stillother embodiments one or more of the four camera pods 500 are mounted inplaces other than the monitor 600.

In alternate embodiments three camera pods are used and positioned atthe top or bottom of the monitor 600. Some embodiments use two camerapods that are positioned at the top or bottom corners of the monitor600. In still other embodiments N camera pods are used, where N isgreater than four (N>4). In this embodiment the N camera pods arepositioned around the outside edge of the monitor 600. In yet otherembodiments there are multiple camera pods positioned behind the monitor600 in order to capture the 3D scene of the environment containing thelocal participant.

V.C. Geometric Proxy Creation

Another part of the capture and creation component 200 is the geometricproxy creation module 245. It should be noted that the geometric proxyis not an avatar or a graphical representation of the user. Instead, thegeometric proxy is a geometric representation of the participant thathas real video painted onto the geometric representation frame by framein order to increase the realism. The module 245 creates a geometricproxy for each of the participants in the conference or meeting. Depthinformation is computed from range data captured by the camera pods 500.Once the depth information is obtained a sparse point cloud is createdfrom depth points contained in the captured depth information. A densedepth point cloud then is generated using known methods and the captureddepth information. In some embodiments a mesh is constructed from thedense point cloud and the geometric proxy is generated from the mesh. Inalternate embodiments the dense point clouds are textured in order togenerate the geometric proxy.

FIG. 7 illustrates an exemplary overview of the creation of a geometricproxy for a single meeting participant. As shown in FIG. 7, RGB data 700is captured from the RGB cameras of the camera pods 500. In addition,depth information 710 is computed from the depth data obtained by thecamera pods 500. The RGB data 700 and the depth information 710 areadded together in order to create the geometric proxy 250 for the singlemeeting participant. This geometric proxy creation is performed for eachof the participants such that each participant has a correspondinggeometric proxy.

V.D. Registration of the 3D Volume and Alignment of the 3D Space

The second component of embodiments of the perspective-correctcommunication window system 100 and method is the scene geometrycomponent 210. This includes both the registration of the 3D volume andthe alignment of the 3D space that the camera pods 500 capture. Thegeneral idea of the scene geometry component 210 is to create relativegeometry between the meeting participants. The desire is to align thescene exactly as if the participants are in the same physical locationand engaged in an in-person conversation.

Embodiments of the system 100 and method create the scene geometry thatis a 3D scene anchored at the capturing environment. In order to achievethis it is desirable to have a precise estimation of the environmentscontaining each of the participants. Once this is obtained thenembodiments of the system 100 and method compute a precise registrationof the monitor with the cameras. This yields an orientation in virtualspace that is aligned with the real world. In other words, the virtualspace is aligned with the real space. This registration and alignment isachieved using known methods. In some embodiments of the system 100 andmethod the calibration is performed at the time of manufacture. In otherembodiments calibration is performed using a reference object in theenvironment.

The scene geometry seeks to create relative geometry between a localparticipant and remote participants. This includes creating eye gaze andconversional geometry as if the participants were in an in-personmeeting. One way in which to get eye gaze and conversational geometrycorrect is to have relative, consistent geometry between theparticipants. In some embodiments this is achieved by using virtualboxes. Specifically, if a box was drawn around the participants in realspace when the participants are in a room together, then these virtualboxes are recreated in a virtual layout to create the scene geometry.The shape of the geometry does not matter as much as its consistencybetween the participants.

Certain input form factors like single monitor or multiple monitors willaffect the optimum layout and scalability of the solution. The scenegeometry also depends on the number of participants. A meeting with twoparticipants (a local participant and a remote participant) is aone-to-one (1:1) scene geometry that is different from the scenegeometry when there are three or more participants. Moreover, as will beseen from the examples below, the scene geometry includes eye gazebetween the participants.

FIG. 8 illustrates an exemplary embodiment of scene geometry betweenparticipants when there are two participants in the meeting. As shown inFIG. 8 this scene geometry for a 1:1 conference 800 includes a firstparticipant 810 and a second participant 820. These participants are notin the same physical location.

In this scene geometry for a 1:1 conference 800, the geometry consistsof two boxes that occupy the spaces in front of the respective monitors(not shown) of the participants 810, 820. A first virtual box 830 isdrawn around the first participant 810 and a second virtual box 840 isdrawn around the second participant 820. Assuming the same size monitorsand consistent setups allows embodiments of the system 100 and method toknow the scene geometry is correct without any manipulation of thecaptured data.

In alternate embodiments of the system 100 and method there are multipleremote participants and the geometry is different from the scenegeometry for a 1:1 conference 800. FIG. 9 illustrates an exemplaryembodiment of the scene geometry between participants when there arethree participants in the meeting. This is the scene geometry for a3-endpoint conference 900. An endpoint is an environment containing aparticipant of the conference or meeting. In a 3-endpoint conferencethere are participants in three different physical locations.

In FIG. 9 the scene geometry for a 3-endpoint conference 900 includesparticipant #1 910, participant #2 920, and participant #3 930 around avirtual round table 935. A virtual box #1 940 is drawn aroundparticipant #1 910, a virtual box #2 950 is drawn around participant #2920, and a virtual box #3 960 is drawn around participant #3 930. Eachof the virtual boxes 940, 950, 960 is placed around the virtual roundtable 935 in an equidistant manner. This creates the scene geometry fora 3-endpoint conference 900.

This scene geometry can be extended for additional endpoints. However,at a certain point, such as when there are 4 to 5 endpoints (dependingon size of screen) with a flat screen, the scene geometry exceeds theability of the capture and render of natural pose positions. In thatcase, in order to preserve conversational geometry while not havingconsistent virtual and physical geometry, embodiments of the system 100and method seek to “pose” participants as they look at one another,exaggerating their movements for people in the call in order to showthem at who they are looking. This, however, can get quite complicatedand can lead to an uncanny valley type of experience.

V.E. Virtual Camera

The scene geometry component 210 also includes a virtual camera. Thevirtual camera defines the perspective projection according to which anovel view of the 3D geometric proxy will be rendered. This allowsembodiments of the system 100 and method to obtain a natural eye gazeand connection between people. One breakdown in current videoconferencing occurs because people are not looking where a camera ispositioned, so that the remote participants in the conference feel asthough the other person is not looking at them. This is unnatural andtypically does not occur in an in-person conversion.

The virtual camera in embodiments of the system 100 and method iscreated using the virtual space from the scene geometry and the 3Dgeometric proxy (having detailed texture information) for eachparticipant. This virtual camera is not bound to the locations of thereal camera pods being used to capture the images. Moreover, someembodiments of the system 100 and method use face tracking (includingeye gaze tracking) to determine where the participants are and wherethey are looking in their virtual space. This allows a virtual camera tobe created based on where a participant is looking in the scene. Thisserves to accurately convey the proper gaze of the participant to otherparticipants and provides them the proper view. Thus, the virtual camerafacilitates natural eye gaze and conversational geometry in theinteraction between meeting participants.

Creating a scene geometry and putting extras in that geometry createthese virtual cameras. From the multiple perspectives obtained by thecamera pods the virtual camera is able to move around the scene geometryand see interpolated views where no real camera exists. For examplethink of the head as a balloon. The front of the balloon will becaptured by a camera pod in front of the balloon and one side of theballoon will be captured by a camera pod on that side of the balloon. Avirtual camera can be created anywhere in between the full front and theside by a composition of images from both camera pods. In other words,the virtual camera view is created as a composition of images from thedifferent cameras covering a particular space.

FIG. 10 illustrates an exemplary embodiment of a virtual camera based onwhere a participant is looking. This can also be thought of as usingvirtual gaze to obtain natural eye gaze. As shown in FIG. 10, themonitor 600 displays the remote participant 610 to a local participant1000. The monitor 600 includes the four camera pods 500. A virtual eyegaze box 1010 is drawn around eyes of the remote participant 1020 andeyes of the local participant 1030. The virtual eye gaze box 1010 islevel such that in virtual space the eyes of the remote participant 1020and eyes of the local participant 1030 are looking at each other.

Some embodiments of the virtual camera use face tracking to improveperformance. Face tracking helps embodiments of the system 100 andmethod change the perspective so that the participants are always facingeach other. Face tracking helps the virtual camera remain level with theeye gaze of the viewer. This mimics how our eyes work during anin-person conversation. The virtual camera interacts with the facetracking to create a virtual viewpoint that has the user lookingstraight at the other participant. In other words, the face tracking isused to change the virtual viewpoint of the virtual camera.

V.F. Depth Through Motion Parallax

The third component of the system 100 and method is the virtualviewpoint component 220. Once the rendered geometric proxies and scenegeometry are transmitted to the participants it is rendered on themonitors of the participants. In order to add realism to the scenedisplayed on the monitor, depth using motion parallax is added toprovide the nuanced changes in view that come when the position ofsomeone viewing something changes.

Motion parallax is added using high-speed head tracking that shifts thecamera view as the viewer's head moves. This creates the illusion ofdepth. FIG. 11 illustrates an exemplary embodiment of providing depththrough motion parallax based on where a viewer is facing. As shown inFIG. 11, the monitor 600 having the four camera pods 500 displays animage of the remote participant 610. Note that in FIG. 11 the remoteparticipant 610 is shown as a dotted-line figure 1100 and a solid-linefigure 1110. The dotted-line figure 1110 illustrates that the remoteparticipant 610 is looking to his left and thus has a firstfield-of-view 1120 that includes a dotted-line participant 1130. Thesolid-line figure 1110 illustrates that the remote participant 610 islooking to his right and thus has a second field-of-view 1140 thatincludes a solid-line participant 1150.

As the remote participant's 610 viewpoint moves side to side hisperspective into the other space changes. This gives the remoteparticipant 610 a different view of the other participants and the room(or environment) in which the other participants are located. Thus, ifthe remote participant moves left, right, up, or down he will see aslightly different view of the participant that the remote participant610 is interacting with and the background behind that person shifts aswell. This gives the scene a sense of depth and gives the people in thescene the sense of volume that they get when talking to someone inperson. The remote participant's viewpoint is tracked using headtracking or a low-latency face tracking technique. Depth through motionparallax dramatically enhances the volume feel while providing fullfreedom of movement since the viewer is not locked to one cameraperspective.

V.G. Multiple Participants at a Single Endpoint

Embodiments of the system 100 and method also include the situationwhere there is more than one participant at an endpoint. The abovetechnique for depth through motion parallax works well for a singleviewer because of the ability to track the viewer and to provide theappropriate view on the monitor based on their viewing angle andlocation. This does not work, however, if there is a second person atthe same endpoint and viewing the same monitor because the monitor canonly provide one scene at a time and it will be locked to one person.This causes the view to be off for the other viewer that is not beingtracked.

There are several ways in which embodiments of the system 100 and methodaddress this issue. In some embodiments monitors are used that providedifferent images to different viewers. In these embodiments the facetracking technique tracks two difference faces and then providesdifferent views to different viewers. In other embodiments the motionparallax is removed and a fixed virtual camera is locked in the centerof the monitor. This creates a sub-standard experience when more thanone participant is at an endpoint. In still other embodiments glassesare worn by each of the multiple participants at the endpoint. Each pairof glasses is used to provide different views. In still otherembodiments the glasses have active shutters on them that show eachwearer different frames from the monitor. The alternating framesdisplayed by the monitor are tuned to each pair of glasses and provideeach viewer the correct image based on the viewer's location.

Another embodiment uses a monitor having multiple viewing angles. FIG.12 illustrates an exemplary embodiment of a technique to handle multipleparticipants at a single endpoint using the monitor having multipleviewing angles. This provides each viewer in front of the monitor with adifferent view of the remote participant 610 and the room behind theremote participant 610.

As shown in FIG. 12, a monitor having a lenticular display 1200 (whichallows multiple viewing angles) and having the four camera pods 500 isdisplaying the remote participant 610. A first viewer 1210 is looking atthe monitor 1200 from the left side of the monitor 1200. The eyes of thefirst viewer 1220 are looking at the monitor 1200 from the left side andhave a left field-of-view 1230 of the monitor 1200. A second viewer 1240is looking at the monitor 1200 from the right side of the monitor 1200.The eyes of the second viewer 1250 are looking at the monitor 1200 fromthe right side and have a right field-of-view 1260. Because of thelenticular display on the monitor 1200, the left field-of-view 1230 andthe right field-of-view 1260 are different. In other words, the firstviewer 1210 and the second viewer 1240 are provided with different viewof the remote participant 610 and the room behind the remote participant610. Thus, even if the first viewer 1210 and the second viewer 1240 wereside by side, they would see different things on the monitor 1200 basedon their viewpoint.

Moreover, although the subject matter has been described in languagespecific to structural features and/or methodological acts, it is to beunderstood that the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Rather, the specific features and acts described above are disclosed asexample forms of implementing the claims.

1. A method for creating an in-person communication experience betweenparticipants in an online meeting, comprising: capturing RGB data anddepth information of the local participant from a plurality of camerapods arranged around a monitor that a local participant is viewing;creating a geometric proxy for the local participant by adding the RGBand the depth information; generating scene geometry based on a numberof participants in the online meeting, including the local participant;rendering the geometric proxy for the local participant and geometricproxies for each of the other participants in the online meeting to eachother in the scene geometry so that the scene geometry is consistentwith an in-person conversation; transmitting the rendered geometricproxies and scene geometry to each of the participants; displaying avirtual viewpoint to the local participant that includes a renderedgeometric proxy for each of the participants along with a backgroundthat is part of the scene geometry; changing the virtual viewpoint basedon a position of the local participant's face such that the localparticipant's view of other participants and the background is dependenton an orientation of the local participant's face; and adding anillusion of depth to the virtual viewpoint using motion parallax tocreate the in-person communication experience between the participantsin the online meeting.
 2. The method of claim 1, further comprising:having the local participant and another participant at the samephysical location and viewing the monitor, where the monitor has alenticular display that allows multiple viewing angles; and having thelocal participant view the monitor from the right side to obtain a rightfield-of-view and having the other participant view the monitor from theleft side to obtain a left field-of-view, the right field-of-view andthe left field-of-view being different.
 3. The method of claim 1,further comprising: having the local participant and another participantat the same physical location and viewing the monitor; using a facetracking technique to track the local participant and the otherparticipant; and providing different views on the monitor to the localparticipant and the other participant based on an orientation of theirfaces.
 4. The method of claim 1, further comprising using a facetracking technique to track the orientation of the local participant'sface.
 5. The method of claim 1, further comprising keeping the virtualviewpoint level with an eye gaze of the local participant using a facetracking technique.
 6. The method of claim 1, wherein said virtualviewpoint is created using a virtual camera that is a composition ofimages from at least two of the plurality of camera pods.
 7. The methodof claim 1, further comprising: using virtual boxes to ensure that aneye gaze and conversational geometry between the participants meetsprescribed characteristics and that to other participants theconversational geometry also meets the prescribed characteristics; anddetermining a number of virtual boxes to use based on the number ofparticipants.
 8. The method of claim 7, further comprising: determiningthat there are two participants in the meeting; and creating a firstvirtual box and a second virtual box that are facing each other.
 9. Themethod of claim 7, further comprising: determining that there are threeparticipants in the meeting; and creating a first virtual box, a secondvirtual box, and a third virtual box that are place around a virtualround table in an equidistant manner.
 10. The method of claim 1, furthercomprising: tracking the position of the local participant's face usinga face tracking technique; and keeping the local participant facing atleast some of the other participants using the position of the localparticipant's face to ensure that the local participant and at least oneof the other participants are always looking straight at each other. 11.A system for creating an in-person communication experience betweenparticipants in an online meeting, comprising: a plurality of camerapods arranged around a monitor that a local participant is viewing; oneor more computing devices, said computing devices being in communicationwith each other via a computer network whenever there is a plurality ofcomputing devices; and a computer program having sub-programs executableby the one or more computing devices, the one or more computing devicesbeing directed by the sub-programs of the computer program to, captureRGB data and depth information of the local participant from theplurality of camera pods, create a geometric proxy for the localparticipant by adding the RGB and the depth information, generate scenegeometry based on a number of participants in the online meeting,including the local participant, render the geometric proxy for thelocal participant and geometric proxies for each of the otherparticipants in the online meeting to each other in the scene geometryso that the scene geometry is consistent with an in-person conversation,transmit the rendered geometric proxies and scene geometry to each ofthe participants, display a virtual viewpoint to the local participantthat includes a rendered geometric proxy for each of the participantsalong with a background that is part of the scene geometry, change thevirtual viewpoint based on a position of the local participant's facesuch that the local participant's view of other participants and thebackground is dependent on an orientation of the local participant'sface, and add an illusion of depth to the virtual viewpoint using motionparallax to create the in-person communication experience between theparticipants in the online meeting.
 12. The system of claim 11, whereinthe local participant and another participant are at the same physicallocation and viewing the monitor, the computer program furthercomprising sub-programs for: using a face tracking technique to trackthe local participant and the other participant; and providing differentviews on the monitor to the local participant and the other participantbased on an orientation of their faces.
 13. The system of claim 11,wherein the computer program further comprising sub-programs for: usinga face tracking technique to track the orientation of the localparticipant's face.
 14. The system of claim 11, wherein the computerprogram further comprising sub-programs for: keeping the virtualviewpoint level with an eye gaze of the local participant using a facetracking technique.
 15. The system of claim 11, wherein said virtualviewpoint is created using a virtual camera that is a composition ofimages from at least two of the plurality of camera pods.
 16. The systemof claim 11, wherein the computer program further comprisingsub-programs for: using virtual boxes to ensure that an eye gaze andconversational geometry between the participants meets prescribedcharacteristics and that to other participants the conversationalgeometry also meets the prescribed characteristics; and determining anumber of virtual boxes to use based on the number of participants. 17.The system of claim 16, wherein the sub-program for determining a numberof virtual boxes comprising sub-programs for: determining that there aretwo participants in the meeting; and creating a first virtual box and asecond virtual box that are facing each other.
 18. The system of claim16, wherein the sub-program for determining a number of virtual boxescomprising sub-programs for: determining that there are threeparticipants in the meeting; and creating a first virtual box, a secondvirtual box, and a third virtual box that are place around a virtualround table in an equidistant manner.
 19. The system of claim 11,wherein the computer program further comprising sub-programs for: usinga face tracking technique to track the position of the localparticipant's face; and keeping the local participant facing at leastsome of the other participants using the position of the localparticipant's face to ensure that the local participant and at least oneof the other participants are always looking straight at each other. 20.A computer storage medium having stored thereon computer-executableinstructions for creating an in-person communication experience betweenparticipants in an online meeting, comprising: capturing RGB data anddepth information of the local participant from a plurality of camerapods arranged around a monitor that a local participant is viewing;creating a geometric proxy for the local participant by adding the RGBand the depth information; generating scene geometry based on a numberof participants in the online meeting, including the local participant;rendering the geometric proxy for the local participant and geometricproxies for each of the other participants in the online meeting to eachother in the scene geometry so that the scene geometry is consistentwith an in-person conversation; transmitting the rendered geometricproxies and scene geometry to each of the participants; displaying avirtual viewpoint to the local participant that includes a renderedgeometric proxy for each of the participants along with a backgroundthat is part of the scene geometry; changing the virtual viewpoint basedon a position of the local participant's face such that the localparticipant's view of other participants and the background is dependenton an orientation of the local participant's face; and adding anillusion of depth to the virtual viewpoint using motion parallax tocreate the in-person communication experience between the participantsin the online meeting.